{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## 自动生成聊天的尝试\n",
        "\n",
        "这个脚本由李鲁鲁开发， 属于[Chat凉宫春日](https://github.com/LC1332/Chat-Haruhi-Suzumiya) \n",
        "\n",
        "用于研究是否能够基于GPT生成大量的对话数据\n",
        "\n",
        "**Chat凉宫春日**是模仿凉宫春日等一系列动漫人物，使用近似语气、个性和剧情聊天的语言模型，\n",
        "\n",
        "<details>\n",
        "  <summary> 由李鲁鲁，冷子昂，闫晨曦，封小洋等开发。 </summary>\n",
        "\n",
        "李鲁鲁发起了项目，并完成了最早的版本，在多个微信群实现了测试。\n",
        "\n",
        "冷子昂参与了早期Gradio的开发，并且参与了后端和前端的选型\n",
        "\n",
        "闫晨曦将李鲁鲁的notebook重构为app.py\n",
        "\n",
        "封小洋进行了中文转日文模型的选型\n",
        "\n",
        "</details>\n",
        "\n",
        "- [x] 对所有数据生成关键字\n",
        "- [x] 对所有对话生成关键字\n",
        "- [x] 尝试一下通过长对话反过来生成句子"
      ],
      "metadata": {
        "id": "jvOODaOndZ3k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "我们来试验一下prompt\n",
        "\n",
        "```\n",
        "提取反引号文本中的关键字Entity，以list的形式输出在一个json中。\n",
        "\n",
        "例子输入:`阿虚:「我无意中听到一件事。」\n",
        "春日:「反正不会是什么重要的事。」`\n",
        "例子输出:`{\"Entity\": [\"不重要的事\",\"阿虚\",\"春日\"]}`\n",
        "\n",
        "例子输入:`阿虚:「你为什么要剪头发啊？」\n",
        "春日:「没什么理由，就是想剪了而已。」`\n",
        "例子输出:`{\"Entity\": [\"剪头发\",\"没什么理由\"]}`\n",
        "\n",
        "输入:`春日:「好想要一台电脑哦！」\n",
        "旁白:春日盘着腿坐在一张从其他社团“拿”来的一张桌子，腿旁是一个小三角锥，上面用汉字写着大大的「团长」二字\n",
        "春日:「在这个资讯化的时代里，连一台电脑都没有，是不行的！」\n",
        "春日:「所以，我会弄一台电脑回来。」\n",
        "阿虚:「弄一台，你是说电脑吗?去哪里弄?你该不会打算去抢电器行吧?」\n",
        "春日:「怎么可能!是更近一点的地方啦!」\n",
        "春日:「拿着这个拍立得。」\n",
        "春日:「给我听好了!现在要告诉你作战计划，你可要按照计划行动喔!千万要好好把握机会。」\n",
        "阿虚:「啊?你又要乱来啦?」\n",
        "春日:「有什么关系！」`\n",
        "```\n",
        "\n",
        "ok这样很好，让我们来组织一下这个脚本\n",
        "\n",
        "```\n",
        "{\"Entity\": [\"电脑\", \"资讯化的时代\", \"弄回来\", \"拍立得\", \"作战计划\", \"按照计划行动\", \"把握机会\", \"乱来\"]}\n",
        "```"
      ],
      "metadata": {
        "id": "a_sc7oxAcqDS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "这次我们考虑用LangChain来做"
      ],
      "metadata": {
        "id": "nF9sS_uqip77"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h7aTQZWXb9DK",
        "outputId": "dde3252f-4e2b-4a01-fb39-94282acea6d5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.0/72.0 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.0/20.0 MB\u001b[0m \u001b[31m67.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m91.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m101.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m969.4/969.4 kB\u001b[0m \u001b[31m75.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m84.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.0/57.0 kB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m288.1/288.1 kB\u001b[0m \u001b[31m18.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.4/75.4 kB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m236.8/236.8 kB\u001b[0m \u001b[31m26.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.5/50.5 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m137.0/137.0 kB\u001b[0m \u001b[31m19.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m144.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m29.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m149.6/149.6 kB\u001b[0m \u001b[31m19.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.1/49.1 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.5/72.5 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "#@title 安装环境\n",
        "! pip -q install openai gradio transformers tiktoken langchain gradio"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 输入OpenAI Token\n",
        "\n"
      ],
      "metadata": {
        "id": "8sS5LjUPixMV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import openai\n",
        "\n",
        "openai.api_key = 'sk-lfrdoJK' # 在这里输入你的OpenAI API Token\n",
        "os.environ[\"OPENAI_API_KEY\"] = openai.api_key "
      ],
      "metadata": {
        "id": "scX_PkEKir2I"
      },
      "execution_count": 54,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "tyTeNFe0jPx7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "KEYWORD_PROMPT = \"\"\"\n",
        "提取反引号文本中的关键字Entity，以list的形式输出在一个json中。\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "rZiWuhDxjQff"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title 定义关键词提取prompt\n",
        "\n",
        "from langchain.chat_models import ChatOpenAI\n",
        "\n",
        "from langchain.prompts.chat import (\n",
        "    ChatPromptTemplate,\n",
        "    SystemMessagePromptTemplate,\n",
        "    AIMessagePromptTemplate,\n",
        "    HumanMessagePromptTemplate,\n",
        ")\n",
        "from langchain.schema import (\n",
        "    AIMessage,\n",
        "    HumanMessage,\n",
        "    SystemMessage\n",
        ")\n",
        "\n",
        "chatModel = ChatOpenAI(temperature=0)\n",
        "\n",
        "def extract_keywords( new_query ):\n",
        "\n",
        "    input1 = \"\"\"阿虚:「我无意中听到一件事。」\n",
        "    春日:「反正不会是什么重要的事。」\"\"\"\n",
        "    output1 = \"\"\"{\"Entity\": [\"不重要的事\",\"阿虚\",\"春日\"]}\"\"\"\n",
        "\n",
        "    input2 = \"\"\"阿虚:「你为什么要剪头发啊？」\n",
        "    春日:「没什么理由，就是想剪了而已。」\"\"\"\n",
        "    output2 = \"\"\"{\"Entity\": [\"剪头发\",\"没什么理由\"]}\"\"\"\n",
        "\n",
        "    messages = [\n",
        "        SystemMessage(content=KEYWORD_PROMPT),\n",
        "        HumanMessage(content=input1),\n",
        "        AIMessage(content=output1),\n",
        "        HumanMessage(content=input2),\n",
        "        AIMessage(content=output2)\n",
        "    ]\n",
        "\n",
        "    messages.append(HumanMessage(content=new_query))\n",
        "\n",
        "    return_msg = chatModel(messages)\n",
        "\n",
        "    response = return_msg.content\n",
        "\n",
        "    return response"
      ],
      "metadata": {
        "id": "_X8aREiOjAsT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "test_input = \"\"\"旁白:春日从其中一个纸袋取出上头印了些手写文字的A4草稿纸。\n",
        "春日:「这是为了让大家认识我们SOS团而特别做的传单。这两百张传单，可是我偷溜进印刷室辛辛苦苦印出来的喔!」\n",
        "旁白:春日将传单分发给了大家。上面写着公告『SOS团创团声明 我们SOS团正大募集这世界上所有不可思议的事。欢迎过去曾经历不可思议事件的人，或是现在正面临不可思议、谜样现象的人，以及有预感不久的将来一定直经历奇幻事件的人踊跃与我们咨询。我们会尽力替你解决问题。不过，普通的不可思议事件恕不受理，一定要让我们觉得相当惊人的不可思议事件才行。敬请注意。电子信箱如下……』\n",
        "春日:「好了，该去发传单了。」\n",
        "阿虚:「去哪里发?」\n",
        "春日:「校门口，现在还有很多学生没回家。」\n",
        "阿虚:「是是是，你说的都对。」\n",
        "春日:「你不用发没关系，实玖瑠跟我去就好了。」\n",
        "阿虚:「什么?」\n",
        "\"\"\"\n",
        "\n",
        "keywords = extract_keywords(test_input)\n",
        "\n",
        "print(keywords)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bUGmSwohkE89",
        "outputId": "6fbcab56-3db2-4ab0-ef2a-caad7c30cf35"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\"Entity\": [\"SOS团\",\"传单\",\"印刷室\",\"大家\",\"不可思议的事\",\"不可思议事件\",\"电子信箱\",\"校门口\",\"学生\",\"实玖瑠\"]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 将所有的经典故事，提取为关键字"
      ],
      "metadata": {
        "id": "MTcL95mfkmGg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title 读取所有文本，存储到titles 和字典title_to_text\n",
        "\n",
        "!git clone https://github.com/LC1332/Prophet-Andrew-Ng\n",
        "\n",
        "prophet_data_folder = '/content/Prophet-Andrew-Ng/haruhi-data'\n",
        "\n",
        "import os\n",
        "\n",
        "import tiktoken\n",
        "\n",
        "enc = tiktoken.get_encoding(\"cl100k_base\")\n",
        "\n",
        "titles = []\n",
        "title_to_text = {}\n",
        "\n",
        "# scan all txt file in prophet_data_folder\n",
        "for file in os.listdir(prophet_data_folder):\n",
        "    if file.endswith('.txt'):\n",
        "        title_name = file[:-4]\n",
        "        titles.append(title_name)\n",
        "\n",
        "        with open(os.path.join(prophet_data_folder, file), 'r') as f:\n",
        "            title_to_text[title_name] = f.read()\n",
        "\n",
        "# report length of each text\n",
        "for title in titles:\n",
        "    print(title, len(enc.encode(title_to_text[title])),end= ' | ')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "id": "Wqb7EDnbkRqT",
        "outputId": "fcb90178-f643-48ab-aff7-1c66c1788f62"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fatal: destination path 'Prophet-Andrew-Ng' already exists and is not an empty directory.\n",
            "从哪儿搞电脑 319 | 电研社初次会面 416 | 介绍其他社员 254 | 日常3 216 | 地球上小小的螺丝钉 993 | 电子邮箱 143 | 最新的电脑 200 | 转学生 286 | 自我介绍 115 | 询问朝仓信息 362 | 初中交往经历 168 | 像普通人一样生活 684 | SOS团起名由来 265 | 朝仓转学 457 | 约翰史密斯 168 | 与朝仓公寓管理员谈话 474 | 兔女郎被老师驱散 444 | 开学第二天 210 | 交往的男生 638 | 社团教室 715 | 最后一名社员 357 | 古泉是男的还是女的 203 | 春日与阿虚 149 | 第一次全员大会 374 | 凉宫春日为何转变 154 | 让阿虚帮忙建社团 287 | 转学生的消息 236 | 奇怪的朝仓 296 | 搞电脑过程 438 | 自己建一个社团就好啦 353 | 无聊的社团 284 | 不重要的事情 38 | 电脑是怎么来的 153 | 拉壮丁 668 | 谁来写网站 193 | 团长设定 201 | 兔女郎的反应 239 | 颜色与星期 473 | 为什么剪头发 43 | 凉宫春日的基础设定 217 | 无聊的日常2 288 | 萌角色的重要性 692 | 春日与有希 101 | 没有灵异事件 665 | 兔女郎 332 | 带上阿虚去朝仓家 394 | 传单 424 | 找管理员借钥匙 115 | "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "keywords = []\n",
        "\n",
        "for title in titles:\n",
        "    text = title_to_text[title]\n",
        "    keyword = extract_keywords(text)\n",
        "    keywords.append(keyword)"
      ],
      "metadata": {
        "id": "DTCrm6D9k5lV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(keywords[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bVxHE4nam3Al",
        "outputId": "e752a441-06b0-4849-cdd2-dc3b43044dc0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\"Entity\": [\"电脑\",\"资讯化的时代\",\"拍立得\",\"作战计划\",\"把握机会\"]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title 打开文件并写入jsonl\n",
        "\n",
        "import json\n",
        "\n",
        "with open(\"all_story_keywords.jsonl\", \"w\", encoding='utf-8') as f:\n",
        "    for keyword in keywords:\n",
        "        try:\n",
        "            keyword_dict = json.loads(keyword)\n",
        "            json.dump(keyword_dict, f, ensure_ascii=False)\n",
        "            f.write('\\n')\n",
        "        except:\n",
        "            print(\"Error: cannot load keyword:\", keyword)\n",
        "            continue"
      ],
      "metadata": {
        "id": "bxVR7RuRlKwx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "这个非常令人满意，让我们把过往用户的对话也提取一下"
      ],
      "metadata": {
        "id": "CIf-YGwnoujg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "save_path = \"/content/drive/MyDrive/GPTData/Haruhi-Lulu/\""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uajwi6gYo4ev",
        "outputId": "155bf8ab-900d-47e6-fe61-4830a5968c61"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title 解析所有对话数据\n",
        "\n",
        "import os\n",
        "import json\n",
        "\n",
        "save_path = \"/content/drive/MyDrive/GPTData/Haruhi-Lulu/\"\n",
        "\n",
        "chat_data = []\n",
        "\n",
        "for file_name in os.listdir(save_path):\n",
        "    # print(file_name)\n",
        "    file_path = os.path.join(save_path, file_name)\n",
        "    with open(file_path, 'r', encoding='utf-8') as f:\n",
        "        data = f.read()\n",
        "        segments = data.split('\\n---\\n')\n",
        "        for i in range(0, len(segments)-1, 2):\n",
        "            query = segments[i]\n",
        "            response = segments[i+1]\n",
        "            role_a = role_b = \"\"\n",
        "            if ':' in query:\n",
        "                role_a, query = query.split(':', 1)\n",
        "            elif '：' in query:\n",
        "                role_a, query = query.split('：', 1)\n",
        "            if ':' in response:\n",
        "                role_b, response = response.split(':', 1)\n",
        "            elif '：' in response:\n",
        "                role_b, response = response.split('：', 1)\n",
        "            chat = {\"role_A\":role_a.strip(), \"role_B\":role_b.strip(), \"query\":query.strip(), \"response\":response.strip()}\n",
        "            chat_data.append(chat)\n",
        "\n",
        "print(json.dumps(chat_data, ensure_ascii=False))\n",
        "\n",
        "cleaned_chat_data = []\n",
        "\n",
        "for i in range(len(chat_data)):\n",
        "    is_duplicate = False\n",
        "    for j in range(i+1, len(chat_data)):\n",
        "        if chat_data[i]['query'] == chat_data[j]['query'] and chat_data[i]['response'] == chat_data[j]['response']:\n",
        "            is_duplicate = True\n",
        "            break\n",
        "    if not is_duplicate:\n",
        "        cleaned_chat_data.append(chat_data[i])\n",
        "\n",
        "chat_datas = cleaned_chat_data"
      ],
      "metadata": {
        "cellView": "form",
        "id": "YShbdSyMmKVA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(chat_datas))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "twdiVKWvpBOG",
        "outputId": "bcefea7e-a0ea-4848-83b1-e57a97570884"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "560\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "chat_datas[0]['key'] = ['new_key']\n",
        "print(chat_datas[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HnsMhVGTpe9v",
        "outputId": "f8d0a708-785e-4134-c5e8-18b38abb05ac"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'role_A': '阿虚', 'role_B': '春日', 'query': '「今天在计算机课上老师教了我写Python!」', 'response': '「哦？Python？那你能不能帮我写一个程序啊？」', 'key': ['new_key']}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def json_chat_2_str( data ):\n",
        "    return str(data['role_A'] + ':' + data['query'] + '\\n' + data['role_B'] + ':' + data['response'])\n",
        "\n",
        "def json_chat_2_str_A( data ):\n",
        "    return str(data['role_A'] + ':' + data['query'])"
      ],
      "metadata": {
        "id": "qmj-UQ3TpDfN"
      },
      "execution_count": 87,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from tqdm import tqdm\n",
        "\n",
        "for chat in tqdm(chat_datas, desc=\"Extracting Keywords\"):\n",
        "    text = json_chat_2_str_A(chat)\n",
        "    response = extract_keywords(text)\n",
        "    chat['key_json'] = response\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SXCSHAnepWUU",
        "outputId": "86d6366a-b617-4276-9078-e535e55ecadd"
      },
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Extracting Keywords:  22%|██▏       | 121/560 [05:51<18:36,  2.54s/it]WARNING:langchain.chat_models.openai:Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID b46e7ec4269d9bde33d15e44475bb12a in your message.).\n",
            "Extracting Keywords:  42%|████▏     | 233/560 [10:37<10:14,  1.88s/it]WARNING:langchain.chat_models.openai:Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 8c0ce81efe68fae15293422ea555c146 in your message.).\n",
            "Extracting Keywords:  95%|█████████▍| 530/560 [24:19<01:15,  2.53s/it]WARNING:langchain.chat_models.openai:Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID da4074f826b2bbe659e95d4f9b4f111b in your message.).\n",
            "Extracting Keywords: 100%|██████████| 560/560 [26:20<00:00,  2.82s/it]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title 保存chat_dats为pkl\n",
        "\n",
        "import pickle\n",
        "fname = '/content/drive/MyDrive/GPTData/haruhi_chat_datas.pkl'\n",
        "with open(fname, 'wb') as f:\n",
        "    pickle.dump(chat_datas, f)\n",
        "\n",
        "def load_chat_datas():\n",
        "    with open(fname, 'rb') as f:\n",
        "        chat_datas = pickle.load(f)\n",
        "\n",
        "    return chat_datas"
      ],
      "metadata": {
        "id": "QP6e90icr99N"
      },
      "execution_count": 95,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(chat_datas[1]['key_json'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2Q4MVB1_pt1H",
        "outputId": "ac465181-2f7a-4087-e7ba-df536c519b51"
      },
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\"Entity\": [\"写程序\",\"什么样的程序\"]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(chat_datas[1].keys())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JFJlqlOS5A-s",
        "outputId": "6048cd9a-4c8a-41a9-b87d-a345cc1a5e00"
      },
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dict_keys(['role_A', 'role_B', 'query', 'response', 'key_json'])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(chat_datas[1])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XZpm2-NJ5cAH",
        "outputId": "0b911637-48a5-48c5-c397-55dc71bdbfb8"
      },
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'role_A': '阿虚', 'role_B': '春日', 'query': '「你想写一个什么样的程序呢？」', 'response': '「我想写一个能够预测未来的程序，可以预测天气、地震、彩票号码等等。」', 'key_json': '{\"Entity\": [\"写程序\",\"什么样的程序\"]}'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "ssgm8E1I5blW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "chat_datas是一个python的list，每个元素的keys为'role_A', 'role_B', 'query', 'response', 'key_json'\n",
        "\n",
        "其中key_json是一个类似的json字符串\n",
        "\n",
        "{\"Entity\": [\"写程序\",\"预测未来\",\"天气\",\"地震\",\"彩票号码\"]}\n",
        "\n",
        "- 遍历chat_datas中的每一个元素，尝试解析key_json字段，如果解析成功，则把其中的Entity存成keyword，不成功则跳过\n",
        "- 对每一个元素，只保存'role_A', 'role_B', 'query', 'response', 'keywords'字段，其他字段都忽略\n",
        "\n",
        "样例element输入:\n",
        "`{'role_A': '阿虚', 'role_B': '春日', 'query': '「你想写一个什么样的程序呢？」', 'response': '「我想写一个能够预测未来的程序，可以预测天气、地震、彩票号码等等。」','other':'to remove',  'key_json': '{\"Entity\": [\"写程序\",\"预测未来\",\"天气\",\"地震\",\"彩票号码\"]}`\n",
        "样例element输出:\n",
        "`{'role_A': '阿虚', 'role_B': '春日', 'query': '「你想写一个什么样的程序呢？」', 'response': '「我想写一个能够预测未来的程序，可以预测天气、地震、彩票号码等等。」', 'keywords': [\"写程序\",\"预测未来\",\"天气\",\"地震\",\"彩票号码\"]}`\n",
        "\n",
        "并把chat_datas中的信息，以jsonl格式保存到all_chat_datas.jsonl"
      ],
      "metadata": {
        "id": "-SWrxllv5KPq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "我们还是争取写个jsonl出来"
      ],
      "metadata": {
        "id": "c2tE93Jx1y5G"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "\n",
        "all_chat_datas = []\n",
        "for element in chat_datas:\n",
        "    try:\n",
        "        key_json = json.loads(element['key_json'])\n",
        "        keywords = key_json['Entity']\n",
        "    except:\n",
        "        print('failed to parse', element['key_json'])\n",
        "        continue\n",
        "    new_element = {\n",
        "        'role_A': element['role_A'],\n",
        "        'role_B': element['role_B'],\n",
        "        'query': element['query'],\n",
        "        'response': element['response'],\n",
        "        'keywords': keywords\n",
        "    }\n",
        "    all_chat_datas.append(new_element)\n",
        "\n",
        "with open('all_chat_datas.jsonl', 'w', encoding='utf-8') as f:\n",
        "    for element in all_chat_datas:\n",
        "        json.dump(element, f, ensure_ascii=False) \n",
        "        f.write('\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FbV7HDDj6nPj",
        "outputId": "179d9276-f15d-4271-d994-689817b2d265"
      },
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "failed to parse 抱歉，我不理解您的意思。请您提供更多上下文信息或者明确您的问题。\n",
            "failed to parse 无法提取关键字Entity，因为这句话中没有明确的实体或关键词。\n",
            "failed to parse 无法提取关键字Entity，因为反引号文本中没有明显的实体或关键词。\n",
            "failed to parse {\"Entity\": []}（无关键字）\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(chat_datas[12]['key_json'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dtBkWmQX11bJ",
        "outputId": "5c2bc7ef-8cda-435f-88ac-29a7f32aded4"
      },
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\"Entity\": [\"李鲁鲁\",\"Haruhi\",\"新转学生\",\"同学\"]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "让我们来尝试生成这个prompt\n",
        "\n",
        "根据keywords的内容补全text，text为对于凉宫春日剧情的一些讨论问题，用一致性的语言风格，根据每行中的json内容，根据keywords中的关键字，补全text的内容。\n",
        "\n",
        "输入:\n",
        "```\n",
        "{\"keywords\":[\"社团\",\"不想来\"]}\n",
        "{\"keywords\":[\"新来\",\"转学\",\"转学生\"]}\n",
        "{\"keywords\":[\"长门\",\"外星人\"]}\n",
        "{\"keywords\":[\"社团\",\"成员\",\"长门\",\"古泉\",\"学姐\"]}\n",
        "```\n",
        "输出:\n",
        "```\n",
        "{\"role\":\"阿虚\", \"text\":\"明天我有点不想来社团活动了\", \"keywords\":[\"社团\",\"不想来\"]}\n",
        "{\"role\":\"李鲁鲁\", \"text\":\"你好呀，Haruhi， 我是新转学来的同学李鲁鲁\", \"keywords\":[\"新来\",\"转学\",\"转学生\"]}\n",
        "{\"role\":\"李鲁鲁\", \"text\":\"什么，你知道长门有希是外星人？\", \"keywords\":[\"长门\",\"外星人\"]}\n",
        "{\"role\":\"阿虚\", \"text\":\"不会吧，怎么只有我和你呢？长门、古一和学姐不是我们社团的成员吗？\", \"keywords\":[\"社团\",\"成员\",\"长门\",\"古泉\",\"学姐\"]}\n",
        "```\n",
        "\n",
        "输入:\n",
        "```\n",
        "{\"keywords\":[\"店员\", \"一览表\", \"名单\", \"相机\"]}\n",
        "{\"keywords\":[ \"转学\", \"国外\", \"灵异事件\"]}\n",
        "{\"keywords\":[\"长门\",\"外星人\"]}\n",
        "{\"keywords\":[\"社团\",\"成员\",\"长门\",\"古泉\",\"学姐\"]}\n",
        "```\n",
        "\n",
        "这个效果还是不错的，GPT的输出为\n",
        "\n",
        "```\n",
        "{\"role\":\"某个顾客\", \"text\":\"店员，你能不能给我一份这个店里的一览表？我想看看名单\", \"keywords\":[\"店员\", \"一览表\", \"名单\"]}\n",
        "{\"role\":\"古泉一树\", \"text\":\"大家好，我是新转学来的同学古泉一树，来自国外\", \"keywords\":[ \"转学\", \"国外\"]}\n",
        "{\"role\":\"长门有希\", \"text\":\"其实我是外星人\", \"keywords\":[\"长门\",\"外星人\"]}\n",
        "{\"role\":\"阿虚\", \"text\":\"不会吧，怎么只有我和你呢？长门、古一和学姐不是我们社团的成员吗？\", \"keywords\":[\"社团\",\"成员\",\"长门\",\"古泉\",\"学姐\"]}\n",
        "```"
      ],
      "metadata": {
        "id": "xxASGPWAuUNh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "那我们后面要做的就是给关键词去重就可以"
      ],
      "metadata": {
        "id": "Au-x1cbKz2py"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "我们现在要做的是，从chat中选取10组关键字，组成few shot"
      ],
      "metadata": {
        "id": "GFR6EflB8dcm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(all_chat_datas[0].keys())\n",
        "print(all_chat_datas[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4ojZB4Hk8mRN",
        "outputId": "cc00db02-0b23-4c43-aabb-ae55fe7e3648"
      },
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dict_keys(['role_A', 'role_B', 'query', 'response', 'keywords'])\n",
            "{'role_A': '阿虚', 'role_B': '春日', 'query': '「今天在计算机课上老师教了我写Python!」', 'response': '「哦？Python？那你能不能帮我写一个程序啊？」', 'keywords': ['计算机课', 'Python']}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "sel_chat_datas是一个python的list of dict，长度为n\n",
        "\n",
        "每个元素的数据格式如下\n",
        "\n",
        "`{'role_A': '阿虚', 'role_B': '春日', 'query': '「今天在计算机课上老师教了我写Python!」', 'response': '「哦？Python？那你能不能帮我写一个程序啊？」', 'keywords': ['计算机课', 'Python', '写程序', '阿虚', '春日']}`\n",
        "\n",
        "我们希望实现一个函数，输入为sel_chat_datas，输出为sample_input, sample_output\n",
        "\n",
        "其中第一个输出 sample_input为n行，每行只保留json中的keywords\n",
        "\n",
        "每行的格式如下\n",
        "\n",
        "`{'keywords': ['计算机课', 'Python', '写程序']}`\n",
        "\n",
        "第二个输出 sample_outout为n行，将role_A重命名为role，query重命名为text，并且保留keywords，每行的格式如下\n",
        "\n",
        "`{'role': '阿虚', 'text': '「今天在计算机课上老师教了我写Python!」', 'keywords': ['计算机课', 'Python', '写程序']}`\n",
        "\n",
        "- 注意这其中每行的keywords中去掉了stop_words中的元素，stop_words = ['春日','阿虚','凉宫','凉宫春日']\n",
        "\n",
        "第三个输出为一个list，包含了去重之后的keywords中的元素\n",
        "\n",
        "请用python实现这个函数"
      ],
      "metadata": {
        "id": "HVD-9DY69zzz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(all_chat_datas))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e9Fiwi2v9s16",
        "outputId": "bc8d4514-9afd-41d8-ce65-ff1344c24b59"
      },
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "556\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "stop_words = ['春日','阿虚','凉宫','凉宫春日']\n",
        "\n",
        "\n",
        "import random \n",
        "\n",
        "n = len(all_chat_datas)\n",
        "sel_all = random.sample(range(1, n+1), 20)\n",
        "sel_ids = sel_all[0:10]\n",
        "sel_test = sel_all[10:]\n",
        "\n",
        "sel_chat_datas = [ all_chat_datas[i] for i in sel_ids ]\n",
        "\n",
        "from typing import List, Dict, Tuple\n",
        "\n",
        "def process_sel_chat_data(sel_chat_datas: List[Dict[str, str]]) -> Tuple[List[Dict], List[Dict], List[str]]:\n",
        "    stop_words = ['春日', '阿虚', '凉宫', '凉宫春日']\n",
        "    sample_input = []\n",
        "    sample_output = []\n",
        "    all_keywords = set()\n",
        "    for element in sel_chat_datas:\n",
        "        keywords = [kw for kw in element['keywords'] if kw not in stop_words]\n",
        "        np.random.shuffle(keywords)\n",
        "        sample_input.append({'keywords': keywords})\n",
        "        output_element = {\n",
        "            'keywords': keywords,\n",
        "            'role': element['role_A'],\n",
        "            'text': element['query'],\n",
        "        }\n",
        "        sample_output.append(output_element)\n",
        "        for kw in keywords:\n",
        "            all_keywords.add(kw)\n",
        "    return sample_input, sample_output, list(all_keywords)\n",
        "\n",
        "\n",
        "sample_input, sample_output, sample_keywords = process_sel_chat_data( sel_chat_datas )\n",
        "\n",
        "sel_test = [ all_chat_datas[i] for i in sel_test ]\n",
        "\n",
        "test_input, _, _ = process_sel_chat_data( sel_test )\n",
        "\n",
        "\n",
        "for sam in sample_input:\n",
        "    print(sam)\n",
        "# print(sample_input,'\\n')\n",
        "\n",
        "print('\\n')\n",
        "\n",
        "for sam in sample_output:\n",
        "    print(sam)\n",
        "\n",
        "print('\\n')\n",
        "\n",
        "for sam in test_input:\n",
        "    print(sam)\n",
        "\n",
        "print('\\n')\n",
        "\n",
        "print(sample_keywords)"
      ],
      "metadata": {
        "id": "mEWOYSpHq-yc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c491a56c-3f55-44dd-c2cf-b444367221d3"
      },
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'keywords': ['如何评价']}\n",
            "{'keywords': ['超能力', '轻功', '御剑飞行']}\n",
            "{'keywords': ['古墓', '进入']}\n",
            "{'keywords': []}\n",
            "{'keywords': ['月球流浪', '求婚']}\n",
            "{'keywords': ['明天', '河边', 'gerry']}\n",
            "{'keywords': ['kiki', '对呀']}\n",
            "{'keywords': ['校园第一节minecraft搭建大赛']}\n",
            "{'keywords': ['破解案子', '投资人', '柯南', '5000亿日元']}\n",
            "{'keywords': ['SOS团', '活动']}\n",
            "\n",
            "\n",
            "{'keywords': ['如何评价'], 'role': '将', 'text': '「如何评价阿虚」'}\n",
            "{'keywords': ['超能力', '轻功', '御剑飞行'], 'role': '阿虚', 'text': '「如果他真的会御剑飞行，那算是一种轻功，还算是一种超能力呢？」'}\n",
            "{'keywords': ['古墓', '进入'], 'role': '柯南', 'text': '「进入古墓」'}\n",
            "{'keywords': [], 'role': 'cj', 'text': '「我也是这样想的！」'}\n",
            "{'keywords': ['月球流浪', '求婚'], 'role': '名人漩涡', 'text': '「那你还愿意和我一起到月球流浪么？还愿意向我求婚么？」'}\n",
            "{'keywords': ['明天', '河边', 'gerry'], 'role': 'gerry', 'text': '「明天去河边」'}\n",
            "{'keywords': ['kiki', '对呀'], 'role': 'kiki', 'text': '「对呀」'}\n",
            "{'keywords': ['校园第一节minecraft搭建大赛'], 'role': '阿虚', 'text': '「我们是不是可以举办校园第一节minecraft搭建大赛」'}\n",
            "{'keywords': ['破解案子', '投资人', '柯南', '5000亿日元'], 'role': '柯南', 'text': '「我这个月破解了好几个案子, 碰到了一个投资人,说要给我们5000亿日元, 我们接受吗」'}\n",
            "{'keywords': ['SOS团', '活动'], 'role': 'Kyon', 'text': '「最近SOS团有什么活动吗」'}\n",
            "\n",
            "\n",
            "{'keywords': ['看法']}\n",
            "{'keywords': ['谁']}\n",
            "{'keywords': ['Kyon']}\n",
            "{'keywords': ['求解', '常微分非齐次线性方程']}\n",
            "{'keywords': ['吉他水平', '长门有希']}\n",
            "{'keywords': ['打羽毛球', '每天晚上', '运动天赋', '工作']}\n",
            "{'keywords': ['下课']}\n",
            "{'keywords': ['你', '还好']}\n",
            "{'keywords': ['上次和朝比奈一起穿兔女郎', '特别的社团活动']}\n",
            "{'keywords': ['永葆青春', '奇异的药水', '艰辛', '奇怪的感情', '永生之路', '帮助', '古墓', '改变基因']}\n",
            "\n",
            "\n",
            "['河边', '如何评价', '5000亿日元', '活动', '破解案子', '进入', '超能力', '明天', '古墓', 'gerry', '柯南', '月球流浪', '轻功', 'SOS团', '对呀', '投资人', '御剑飞行', 'kiki', '求婚', '校园第一节minecraft搭建大赛']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "根据keywords的内容补全text，text为对于凉宫春日剧情的一些讨论问题，用一致性的语言风格，根据每行中的json内容，根据keywords中的关键字，补全text的内容。\n",
        "\n",
        "输入:\n",
        "```js\n",
        "{'keywords': ['冷子昂', '你好', '有什么事吗']}\n",
        "{'keywords': ['班里', '亚丝娜', '同学', '个性', '女孩子', '好朋友', '光线照射', '情况', '线索', '证据', '调查']}\n",
        "{'keywords': ['miku', '你好', '谁', '见过']}\n",
        "{'keywords': ['英文名', 'Cosmic Exploration Club', '缩写', 'CEC', '国际化', '记忆']}\n",
        "{'keywords': ['123', '奇怪的话']}\n",
        "{'keywords': ['不可能', '弟弟', '搞错了人', '宫热']}\n",
        "{'keywords': ['普通人', '高中生', '超能力', '神秘身份', '未知领域', '科学', '理性', '神秘', '超自然', '问题', '疑虑', '交流', '互相猜疑', '攻击']}\n",
        "{'keywords': ['羽毛球', '队友关系', '任务', '个人感情', '工作', '社团活动', '电影']}\n",
        "{'keywords': ['长门有希', '外星人', '超能力', '思维方式', '行为方式', '纯真', '善良', 'SOS团', '有趣', '神秘']}\n",
        "{'keywords': ['约翰史密斯', '名字']}\n",
        "```\n",
        "\n",
        "输出:\n",
        "```js\n",
        "{'keywords': ['如何评价'], 'role': '将', 'text': '「如何评价阿虚」'}\n",
        "{'keywords': ['超能力', '轻功', '御剑飞行'], 'role': '阿虚', 'text': '「如果他真的会御剑飞行，那算是一种轻功，还算是一种超能力呢？」'}\n",
        "{'keywords': ['古墓', '进入'], 'role': '柯南', 'text': '「进入古墓」'}\n",
        "{'keywords': [], 'role': 'cj', 'text': '「我也是这样想的！」'}\n",
        "{'keywords': ['月球流浪', '求婚'], 'role': '名人漩涡', 'text': '「那你还愿意和我一起到月球流浪么？还愿意向我求婚么？」'}\n",
        "{'keywords': ['明天', '河边', 'gerry'], 'role': 'gerry', 'text': '「明天去河边」'}\n",
        "{'keywords': ['kiki', '对呀'], 'role': 'kiki', 'text': '「对呀」'}\n",
        "{'keywords': ['校园第一节minecraft搭建大赛'], 'role': '阿虚', 'text': '「我们是不是可以举办校园第一节minecraft搭建大赛」'}\n",
        "{'keywords': ['破解案子', '投资人', '柯南', '5000亿日元'], 'role': '柯南', 'text': '「我这个月破解了好几个案子, 碰到了一个投资人,说要给我们5000亿日元, 我们接受吗」'}\n",
        "{'keywords': ['SOS团', '活动'], 'role': 'Kyon', 'text': '「最近SOS团有什么活动吗」'}\n",
        "```\n",
        "\n",
        "输入:\n",
        "```js\n",
        "{'keywords': ['看法']}\n",
        "{'keywords': ['谁']}\n",
        "{'keywords': ['Kyon']}\n",
        "{'keywords': ['求解', '常微分非齐次线性方程']}\n",
        "{'keywords': ['吉他水平', '长门有希']}\n",
        "{'keywords': ['打羽毛球', '每天晚上', '运动天赋', '工作']}\n",
        "{'keywords': ['下课']}\n",
        "{'keywords': ['你', '还好']}\n",
        "{'keywords': ['上次和朝比奈一起穿兔女郎', '特别的社团活动']}\n",
        "{'keywords': ['永葆青春', '奇异的药水', '艰辛', '奇怪的感情', '永生之路', '帮助', '古墓', '改变基因']}\n",
        "```\n",
        "\n",
        "GPT的输出:\n",
        "```js\n",
        "{'keywords': ['看法'], 'role': 'cj', 'text': '「你们对这个问题有什么看法吗？」'}\n",
        "{'keywords': ['谁'], 'role': '阿虚', 'text': '「你说的是谁？」'}\n",
        "{'keywords': ['Kyon'], 'role': '长门有希', 'text': '「Kyon，你想说什么？」'}\n",
        "{'keywords': ['求解', '常微分非齐次线性方程'], 'role': '小兰', 'text': '「求解那个常微分非齐次线性方程还是挺难的呢」'}\n",
        "{'keywords': ['吉他水平', '长门有希'], 'role': '长门有希', 'text': '「你们知道吗，长门有希的吉他水平其实非常高！」'}\n",
        "{'keywords': ['打羽毛球', '每天晚上', '运动天赋', '工作'], 'role': '小美', 'text': '「我每天晚上都要打羽毛球，但是我又有很多工作要做，真的是很累啊！不过我相信我自己的运动天赋，会一直保持好的状态的！」'}\n",
        "{'keywords': ['下课'], 'role': 'cj', 'text': '「好了，下课了！」'}\n",
        "{'keywords': ['你', '还好'], 'role': '阿虚', 'text': '「你觉得我还好吗？」'}\n",
        "{'keywords': ['上次和朝比奈一起穿兔女郎', '特别的社团活动'], 'role': '小兰', 'text': '「上次和朝比奈一起穿兔女郎的时候，真的是特别的社团活动啊」'}\n",
        "{'keywords': ['永葆青春', '奇异的药水', '艰辛', '奇怪的感情', '永生之路', '帮助', '古墓', '改变基因'], 'role': '柯南', 'text': '「永葆青春、永生之路，这是一场充满艰辛和奇怪感情的旅程，但是只要我们帮助改变基因就可以了，去探索那个古墓吧！」'}\n",
        "```"
      ],
      "metadata": {
        "id": "-0lOP7--E-k7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "这个notebook有点乱了，因为我们已经把关键词都保存到了两个jsonl中\n",
        "\n",
        "all_chat_datas.jsonl和 all_story_keywords.jsonl\n",
        "\n",
        "让我们重启一个notebook重新开始"
      ],
      "metadata": {
        "id": "9pbZGTM4VFxA"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "K5NDzJP3EIl1"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}